{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Using Amazon Elastic Inference with a pre-trained TensorFlow Serving model on SageMaker\n",
    "\n",
    "This notebook demonstrates how to enable and use Amazon Elastic Inference with our predefined SageMaker TensorFlow Serving containers.\n",
    "\n",
    "Amazon Elastic Inference (EI) is a resource you can attach to your Amazon EC2 instances to accelerate your deep learning (DL) inference workloads. EI allows you to add inference acceleration to an Amazon SageMaker hosted endpoint or Jupyter notebook for a fraction of the cost of using a full GPU instance. For more information please visit: https://docs.aws.amazon.com/sagemaker/latest/dg/ei.html\n",
    "\n",
    "This notebook's main objective is to show how to create an endpoint, backed by an Elastic Inference, to serve our pre-trained TensorFlow Serving model for predictions. With a more efficient cost per performance, Amazon Elastic Inference can prove to be useful for those looking to use GPUs for higher inference performance at a lower cost.\n",
    "\n",
    "1. [The model](#The-model)\n",
    "1. [Setup role for SageMaker](#Setup-role-for-SageMaker)\n",
    "1. [Load the TensorFlow Serving Model on Amazon SageMaker using Python SDK](#Load-the-TensorFlow-Serving-Model-on-Amazon-SageMaker-using-Python-SDK)\n",
    "1. [Deploy the trained Model to an Endpoint with EI](#Deploy-the-trained-Model-to-an-Endpoint-with-EI)\n",
    "    1. [Using EI with a SageMaker notebook instance](#Using-EI-with-a-SageMaker-notebook-instance)\n",
    "    1. [Invoke the Endpoint to get inferences](#Invoke-the-Endpoint-to-get-inferences)\n",
    "    1. [Delete the Endpoint](#Delete-the-Endpoint)\n",
    "\n",
    "If you are familiar with SageMaker and already have a trained model, skip ahead to the [Deploy the trained Model to an Endpoint with an attached EI accelerator](#Deploy-the-trained-Model-to-an-Endpoint-with-an-attached-EI-accelerator)\n",
    "\n",
    "For this example, we will use the SageMaker Python SDK, which helps deploy your models to train and host in SageMaker. In this particular example, we will be interested in only the hosting portion of the SDK.\n",
    "\n",
    "1. Set up our pre-trained model for consumption in SageMaker\n",
    "2. Host the model in an endpoint with EI\n",
    "3. Make a sample inference request to the model\n",
    "4. Delete our endpoint after we're done using it\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The model\n",
    "\n",
    "The pre-trained model we will be using for this example is a NCHW ResNet-50 model from the [official Tensorflow model Github repository](https://github.com/tensorflow/models/tree/master/official/resnet#pre-trained-model). For more information in regards to deep residual networks, please check [here](https://github.com/tensorflow/models/tree/master/official/resnet). It isn't a requirement to train our model on SageMaker to use SageMaker for serving our model.\n",
    "\n",
    "SageMaker expects our models to be compressed in a tar.gz format in S3. Thankfully, our model already comes in that format. The predefined TensorFlow Serving containers use REST API for handling inferences, for more informationm, please see [Deploying to TensorFlow Serving Endpoints](https://github.com/aws/sagemaker-python-sdk/blob/master/src/sagemaker/tensorflow/deploying_tensorflow_serving.rst#making-predictions-against-a-sagemaker-endpoint).\n",
    "\n",
    "To host our model for inferences in SageMaker, we need to first upload the SavedModel to S3. This can be done through the AWS console or AWS command line.\n",
    "\n",
    "For this example, the SavedModel object will already be hosted in a public S3 bucket owned by SageMaker."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "import boto3\n",
    "\n",
    "# use the region-specific saved model object\n",
    "region = boto3.Session().region_name\n",
    "saved_model = 's3://sagemaker-sample-data-{}/tensorflow/model/resnet/resnet_50_v2_fp32_NCHW.tar.gz'.format(region)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup role for SageMaker\n",
    "\n",
    "Let's start by creating a SageMaker session and specifying the IAM role arn used to give hosting access to your model. See the [documentation](https://docs.aws.amazon.com/sagemaker/latest/dg/sagemaker-roles.html) for how to create these. Note, if more than one role is required for notebook instances, training, and/or hosting, please replace the `sagemaker.get_execution_role()` with a the appropriate full IAM role arn string(s)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "isConfigCell": true
   },
   "outputs": [],
   "source": [
    "import sagemaker\n",
    "\n",
    "role = sagemaker.get_execution_role()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load the TensorFlow Serving Model on Amazon SageMaker using Python SDK\n",
    "\n",
    "We can use the SageMaker Python SDK to load our pre-trained TensorFlow Serving model for hosting in SageMaker for predictions.\n",
    "\n",
    "There are a few parameters that our TensorFlow Serving Model is expecting.\n",
    "1. `model_data` - The S3 location of a model tar.gz file to load in SageMaker\n",
    "2. `role` - An IAM role name or ARN for SageMaker to access AWS resources on your behalf.\n",
    "3. `framework_version` - TensorFlow Serving version you want to use for handling your inference request .\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sagemaker.tensorflow.serving import Model\n",
    "\n",
    "tensorflow_model = Model(model_data=saved_model,\n",
    "                         role=role,\n",
    "                         framework_version='1.13')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Deploy the trained Model to an Endpoint with an attached EI accelerator\n",
    "\n",
    "The `deploy()` method creates an endpoint which serves prediction requests in real-time.\n",
    "\n",
    "The only change required for utilizing EI with our SageMaker TensorFlow Serving containers only requires providing an `accelerator_type` parameter, which determines which type of EI accelerator to attach to your endpoint. The supported types of accelerators can be found here: https://aws.amazon.com/sagemaker/pricing/instance-types/\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "%%time\n",
    "predictor = tensorflow_model.deploy(initial_instance_count=1,\n",
    "                                    instance_type='ml.m4.xlarge',\n",
    "                                    accelerator_type='ml.eia1.medium')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Using EI with a SageMaker notebook instance\n",
    "\n",
    "There is also the ability to utilize an EI accelerator attached to your local SageMaker notebook instance. This can be done by modifying `instance_type` to `local` and `accelerator_type` to `local_sagemaker_notebook`. For more information, please reference: https://github.com/awslabs/amazon-sagemaker-examples/blob/master/sagemaker-python-sdk/tensorflow_iris_dnn_classifier_using_estimators/tensorflow_iris_dnn_classifier_using_estimators_elastic_inference_local.ipynb"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#  Invoke the Endpoint to get inferences\n",
    "\n",
    "Invoking prediction:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "%%time\n",
    "import numpy as np\n",
    "random_input = np.random.rand(1, 1, 3, 3)\n",
    "\n",
    "prediction = predictor.predict({'inputs': random_input.tolist()})\n",
    "\n",
    "print(prediction)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Delete the Endpoint\n",
    "\n",
    "After you have finished with this example, remember to delete the prediction endpoint to release the instance(s) associated with it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(predictor.endpoint)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sagemaker\n",
    "\n",
    "predictor.delete_endpoint()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "conda_tensorflow_p27",
   "language": "python",
   "name": "conda_tensorflow_p27"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.15"
  },
  "notice": "Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved.  Licensed under the Apache License, Version 2.0 (the \"License\"). You may not use this file except in compliance with the License. A copy of the License is located at http://aws.amazon.com/apache2.0/ or in the \"license\" file accompanying this file. This file is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License."
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
